Fourier Ptychographic Microscopy Research Articles

Multimodal 2D and 3D microscopic mapping of growth cartilage by computational imaging techniques - a short review including new research.

Being able to image the microstructure of growth cartilage is important for understanding the onset and progression of diseases such as osteochondrosis and osteoarthritis, as well as for developing new treatments and implants. Studies of cartilage using conventional optical brightfield microscopy rely heavily on histological staining, where the added chemicals provide tissue-specific colours. Other microscopy contrast mechanisms include polarization, phase- and scattering contrast, enabling non-stained or "label-free" imaging that significantly simplifies the sample preparation, thereby also reducing the risk of artefacts. Traditional high-performance microscopes tend to be both bulky and expensive. Computational imaging denotes a range of techniques where computers with dedicated algorithms are used as an integral part of the image formation process. Computational imaging offers many advantages like 3D measurements, aberration correction and quantitative phase contrast, often combined with comparably cheap and compact hardware. X-ray microscopy is also progressing rapidly, in certain ways trailing the development of optical microscopy. In this study, we first briefly review the structures of growth cartilage and relevant microscopy characterization techniques with an emphasis on FPM and advanced X-ray microscopies. We next demonstrate with our own results computational imaging through Fourier ptychographic microscopy (FPM) and compare the images with hematoxylin eosin and saffron (HES)-stained histology. Zernike phase contrast, and the nonlinear optical microscopy techniques of second harmonic generation (SHG) and two-photon excitation fluorescence (TPEF) are reported. X-ray attenuation-, phase- and diffraction-contrast computed tomography (CT) images of the very same sample are also presented for comparisons. Future perspectives on the links to artificial intelligence, dynamic studies and in vivo possibilities conclude the article.

Momentum Fourier ptychographic topography

Reflective Fourier Ptychographic Microscopy (FPM) has demonstrated the potential as a technology for semiconductor metrology and inspection applications, offering both a wide field-of-view (FOV) and high resolution with nanoscale height reconstruction accuracy. However, to enhance resolution, the wavelength of the illumination source has been shortened to the ultraviolet (UV) range, leading to a significant decrease in height calculation accuracy due to low signal-to-noise issues. Additionally, the time-consuming aspect persists as a challenge. To address these issues, we introduce Momentum Fourier Ptychographic Topography (MFPT). MFPT can overcome the problem of getting trapped in local minima due to noise, and it does not require an iterative process of total variant regularization, allowing for significant computational time savings. Using UV illumination with MFPT, we measure a United States Air Force (USAF) 1951 target and two step-height samples, providing evidence that MFPT is robust to noise and accurately reconstructs nanoscale height. As a result, utilizing UV illumination with MFPT allows us to achieve a resolution of 173 nm, and we can rapidly and accurately calculate the height of step-height samples with nominal heights of 30 nm and 70 nm.

Depth-of-field extended Fourier ptychographic microscopy without defocus distance priori

Depth-of-field extended Fourier ptychographic microscopy without defocus distance priori

Linear-space-variant model for Fourier ptychographic microscopy.

Fourier ptychographic microscopy (FPM) needs to realize well-accepted reconstruction by image segmentation and discarding problematic data due to artifacts caused by vignetting. However, the imaging results have long suffered from uneven color blocks and the consequent digital stitching artifacts, failing to bring satisfying experiences to researchers and users over the past decade since the invention of FPM. In fact, the fundamental reason for vignetting artifacts lies in that the acquired data does not match the adopted linear-space-invariant (LSI) forward model, i.e., the actual object function is modulated by a quadratic phase factor during data acquisition, which has been neglected in the advancement of FPM. In this Letter, we rederive a linear-space-variant (LSV) model for FPM and design the corresponding loss function for FPM, termed LSV-FPM. Utilizing LSV-FPM for optimization enables the efficient removal of wrinkle artifacts caused by vignetting in the reconstruction results, without the need of segmenting or discarding images. The effectiveness of LSV-FPM is validated through data acquired in both 4f and finite conjugate single-lens systems.

FPM-WSI: Fourier ptychographic whole slide imaging via feature-domain backdiffraction

Fourier ptychographic microscopy (FPM) theoretically provides a solution to the trade-off between spatial resolution and field of view (FOV), and has promising prospects in digital pathology. However, block reconstruction and then stitching has become an unavoidable procedure for reconstruction of large FOV due to vignetting artifacts. This introduces digital stitching artifacts, as the existing image-domain optimization algorithms are highly sensitive to systematic errors. Such obstacles significantly impede the advancement and practical implementation of FPM, explaining why, despite a decade of development, FPM has not gained widespread recognition in the field of biomedicine. We report a feature-domain FPM (FD-FPM) based on the structure-aware forward model to realize stitching-free, full-FOV reconstruction. The loss function is uniquely formulated in the feature domain of images, which bypasses the troublesome vignetting effect and algorithmic vulnerability via feature-domain backdiffraction. Through massive simulations and experiments, we show that FD-FPM effectively eliminates vignetting artifacts for full-FOV reconstruction, and still achieves impressive reconstructions despite the presence of various systematic errors. We also found it has great potential in recovering the data with a lower spectrum overlapping rate, and in realizing digital refocusing without a prior defocus distance. With FD-FPM, we achieved full-color and high-throughput imaging (4.7 mm diameter FOV, 336 nm resolution in the blue channel) free of blocking-and-stitching procedures on a self-developed Fourier ptychographic microscopy whole slide imaging platform. The reported FD-FPM shows the value of FPM for various experimental circumstances, and offers physical insights useful for the developments of models for other computational imaging techniques. The reported platform demonstrates high-quality, high-speed imaging and low cost, and could find applications in many fields of biomedical research, as well as in clinical applications.

Multi-scale fractal Fourier Ptychographic microscopy to assess the dose-dependent impact of copper pollution on living diatoms.

Accumulation of bioavailable heavy metals in aquatic environment poses a serious threat to marine communities and human health due to possible trophic transfers through the food chain of toxic, non-degradable, exogenous pollutants. Copper (Cu) is one of the most spread heavy metals in water, and can severely affect primary producers at high doses. Here we show a novel imaging test to assay the dose-dependent effects of Cu on live microalgae identifying stress conditions when they are still capable of sustaining a positive growth. The method relies on Fourier Ptychographic Microscopy (FPM), capable to image large field of view in label-free phase-contrast mode attaining submicron lateral resolution. We uniquely combine FPM with a new multi-scale analysis method based on fractal geometry. The system is able to provide ensemble measurements of thousands of diatoms in the liquid sample simultaneously, while ensuring at same time single-cell imaging and analysis for each diatom. Through new image descriptors, we demonstrate that fractal analysis is suitable for handling the complexity and informative power of such multiscale FPM modality. We successfully tested this new approach by measuring how different concentrations of Cu impact on Skeletonema pseudocostatum diatom populations isolated from the Sarno River mouth.

LED-based temporal variant noise model for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a technique to reconstruct a high-resolution image from a set of low-resolution images captured with different illumination angles, which is susceptible to ambient noise, system noise, and weak currents when acquiring large-angle images, especially dark field images. To effectively address the noise problem, we propose an adaptive denoising algorithm based on a LED-based temporal variant noise model. Taking the results of blank slide samples as the reference value of noise, and analyzing the distribution of noise, we establish a statistical model for temporal variant noise, describing the relationship between temporal noise and LED spatial location. Based on this model, Gaussian denoising parameters are selected to adaptively denoise the images with different locations, with which high-resolution images can be reconstructed. Compared with other methods, the experimental results show that the proposed method effectively suppresses the noise, recovers more image details, increases the image contrast, and obtains better visual effects. Meanwhile, better objective evaluation also mirrors the advantages of the proposed algorithms.

Classifying breast cancer and fibroadenoma tissue biopsies from paraffined stain-free slides by fractal biomarkers in Fourier Ptychographic Microscopy

Breast cancer is one of the most spread and monitored pathologies in high-income countries. After breast biopsy, histological tissue is stored in paraffin, sectioned and mounted. Conventional inspection of tissue slides under benchtop light microscopes involves paraffin removal and staining, typically with H&E. Then, expert pathologists are called to judge the stained slides. However, paraffin removal and staining are operator-dependent, time and resources consuming processes that can generate ambiguities due to non-uniform staining. Here we propose a novel method that can work directly on paraffined stain-free slides. We use Fourier Ptychography as a quantitative phase-contrast microscopy method, which allows accessing a very wide field of view (i.e., mm2) in one single image while guaranteeing high lateral resolution (i.e., 0.5 µm). This imaging method is multi-scale, since it enables looking at the big picture, i.e. the complex tissue structure and connections, with the possibility to zoom-in up to the single-cell level. To handle this informative image content, we introduce elements of fractal geometry as multi-scale analysis method. We show the effectiveness of fractal features in describing and classifying fibroadenoma and breast cancer tissue slides from ten patients with very high accuracy. We reach 94.0 ± 4.2% test accuracy in classifying single images. Above all, we show that combining the decisions of the single images, each patient’s slide can be classified with no error. Besides, fractal geometry returns a guide map to help pathologist to judge the different tissue portions based on the likelihood these can be associated to a breast cancer or fibroadenoma biomarker. The proposed automatic method could significantly simplify the steps of tissue analysis and make it independent from the sample preparation, the skills of the lab operator and the pathologist.

High-resolution display screen as programmable illumination for Fourier ptychography

We introduce a new programmable illumination strategy for Fourier ptychographic microscopy (FPM), utilizing a high-resolution organic light-emitting diode (OLED) display screen at the vicinity of the sample to achieve super-resolution and quantitative phase imaging of biological samples. High-density pixels in the OLED screen allow for the angle-scanning illumination required for FP by placing the sample slide directly on the screen, where multiple images of the sample are captured while scanning the pixels on the screen. We discuss the design of the OLED-FP illumination and the corresponding image reconstruction strategies and experimentally validate the super-resolution and phase imaging capabilities of FP using a smartphone screen as illumination. Further, we extend our method to patterned illumination strategies, which multiplexes multiple illumination pixels to achieve full-field FP imaging with a reduced number of measurements, where we identify an optimal illumination pattern and density to maximize the imaging speed without sacrificing the image quality. Our approach offers a simple and compact programmable illuminator that can effectively transform any microscope into a compact FPM with resolution improvement and phase imaging capabilities.

Fourier Ptychographic Neural Network Combined with Zernike Aberration Recovery and Wirtinger Flow Optimization.

Fourier ptychographic microscopy, as a computational imaging method, can reconstruct high-resolution images but suffers optical aberration, which affects its imaging quality. For this reason, this paper proposes a network model for simulating the forward imaging process in the Tensorflow framework using samples and coherent transfer functions as the input. The proposed model improves the introduced Wirtinger flow algorithm, retains the central idea, simplifies the calculation process, and optimizes the update through back propagation. In addition, Zernike polynomials are used to accurately estimate aberration. The simulation and experimental results show that this method can effectively improve the accuracy of aberration correction, maintain good correction performance under complex scenes, and reduce the influence of optical aberration on imaging quality.

Computational label-free microscope through a custom-built high-throughput objective lens and Fourier ptychography

Label-free microscopy directly images live samples without fluorescent markers, providing a panoramic view of biological structures and functions. However, pursuing high-throughput and high-content live cell imaging requires both higher spatial resolution and larger field of view, which are limited by the optical system. In this study, we custom-built a 5▪/0.35 objective with a field number of 28 mm, providing a space-bandwidth product of 34 megapixels, which is four times greater than that of commercial high-throughput objectives such as the Nikon Lambda series (4▪/0.2, 10▪/0.45, 20▪/0.75). Furthermore, we incorporated Fourier ptychography microscopy (FPM) into the system, achieving a synthetic numerical aperture of 0.72, suitable for digital pathology and long-term live cell observation. To improve the FPM performance, we proposed a phantom-based calibration method for quantitative correction of illumination angle errors. Additionally, this method can also serve as an initial step for the on-line calibration. The remarkable capabilities of our 5▪/0.35 objective have been demonstrated, as well as the effectiveness of computational label-free microscopy. Furthermore, by combining our original FPM with more advanced hardware and algorithms, our FPM can achieve higher performance.

A Virtual Staining Method Based on Self-Supervised GAN for Fourier Ptychographic Microscopy Colorful Imaging

Fourier ptychographic microscopy (FPM) is a computational imaging technology that has endless vitality and application potential in digital pathology. Colored pathological image analysis is the foundation of clinical diagnosis, basic research, and most biomedical problems. However, the current colorful FPM reconstruction methods are time-inefficient, resulting in poor image quality due to optical interference and reconstruction errors. This paper combines coloring and FPM to propose a self-supervised generative adversarial network (GAN) for FPM color reconstruction. We design a generator based on the efficient channel residual (ECR) block to adaptively obtain efficient cross-channel interaction information in a lightweight manner, and we introduce content-consistency loss to learn the high-frequency information of the image and improve the image quality of the staining. Furthermore, the effectiveness of our proposed method is demonstrated through objective indicators and visual evaluations.

Fourier Ptychographic Microscopy 10 Years on: A Review.

Fourier ptychographic microscopy (FPM) emerged as a prominent imaging technique in 2013, attracting significant interest due to its remarkable features such as precise phase retrieval, expansive field of view (FOV), and superior resolution. Over the past decade, FPM has become an essential tool in microscopy, with applications in metrology, scientific research, biomedicine, and inspection. This achievement arises from its ability to effectively address the persistent challenge of achieving a trade-off between FOV and resolution in imaging systems. It has a wide range of applications, including label-free imaging, drug screening, and digital pathology. In this comprehensive review, we present a concise overview of the fundamental principles of FPM and compare it with similar imaging techniques. In addition, we present a study on achieving colorization of restored photographs and enhancing the speed of FPM. Subsequently, we showcase several FPM applications utilizing the previously described technologies, with a specific focus on digital pathology, drug screening, and three-dimensional imaging. We thoroughly examine the benefits and challenges associated with integrating deep learning and FPM. To summarize, we express our own viewpoints on the technological progress of FPM and explore prospective avenues for its future developments.

Dual-wavelength Fourier ptychographic microscopy for topographic measurement.

Topographic measurements of micro- or nanostructures are essential in cutting-edge scientific disciplines such as optical communications, metrology, and structural biology. Despite the advances in surface metrology, measuring micron-scale steps with wide field of view (FOV) and high-resolution remains difficult. This study demonstrates a dual-wavelength Fourier ptychographic microscopy for high-resolution topographic measurement across a wide FOV using an aperture scanning structure. This structure enables the capture of a three-dimensional (3D) sample's scattered field with two different wavelength lasers, thus allowing the axial measurement range growing from nano- to micro-scale with enhanced lateral resolution. To suppress the unavoidable noises and artifacts caused by temporal coherence, system vibration, etc., a total variation (TV) regularization algorithm is introduced for phase retrieval. A blazed grating with micron-scale steps is used as the sample to validate the performance of our method. The agreement between the high-resolution reconstructed topography with our method and that with atomic force microscopy verified the effectiveness. Meanwhile, numerical simulations suggest that the method has the potential to characterize samples with high aspect-ratio steps.

Hyperuniform illumination subsampling method for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a novel computational microscopy technique, where Angular-varying LED illumination and phase retrieval techniques are used to stitch a wide field-of-view (FOV) and high-resolution (HR) image. However, the imaging process of FPM needs long acquisition and reconstruction time, which limits the application of FPM in dynamic imaging. Here, we propose a hyperuniform illumination subsampling method to generate 2D hyperuniform LED illumination patterns with grid and collision distance constraints, termed HUD-FPM, to improve FPM imaging efficiency. The corresponding LED element can be lighted up in sequence, and the low-resolution (LR) images can be captured and reconstructed. We verify the effectiveness of the proposed method through simulations and experiments. The proposed method can achieve reconstruction quality close to tranditional sampling FPM while reducing the number of image captures by about 67.2%. Images reconstructed through hyperuniform illumination mode have lower root mean square error and exhibit finer details than commonly used sparse sampling methods at high frequencies. Further, the same conclusion can be also applied to the non-uniform and high-NA quasi-dome LED illuminator design.

Single-shot Fourier ptychographic microscopy with isotropic lateral resolution via polarization-multiplexed LED illumination.

Fourier ptychographic microscopy (FPM) has emerged as a new wide-field and high-resolution computational imaging technique in recent years. To ensure data redundancy for a stable convergence solution, conventional FPM requires dozens or hundreds of raw images, increasing the time cost for both data collection and computation. Here, we propose a single-shot Fourier ptychographic microscopy with isotropic lateral resolution via polarization-multiplexed LED illumination, termed SIFPM. Three LED elements covered with 0°/45°/135° polarization films, respectively, are used to provide numerical aperture-matched illumination for the sample simultaneously. Meanwhile, a polarization camera is utilized to record the light field distribution transmitted through the sample. Based on weak object transfer functions, we first obtain the amplitude and phase estimations of the sample by deconvolution, and then we use them as the initial guesses of the FPM algorithm to refine the accuracy of reconstruction. We validate the complex sample imaging performance of the proposed method on quantitative phase target, unstained and stained bio-samples. These results show that SIFPM can realize quantitative imaging for general samples with the resolution of the incoherent diffraction limit, permitting high-speed quantitative characterization for cells and tissues.

Quantitative real-time phase microscopy for extended depth-of-field imaging based on the 3D single-shot differential phase contrast (ssDPC) imaging method.

Optical diffraction tomography (ODT) is a promising label-free imaging method capable of quantitatively measuring the three-dimensional (3D) refractive index distribution of transparent samples. In recent years, partially coherent ODT (PC-ODT) has attracted increasing attention due to its system simplicity and absence of laser speckle noise. Quantitative phase imaging (QPI) technologies represented by Fourier ptychographic microscopy (FPM), differential phase contrast (DPC) imaging and intensity diffraction tomography (IDT) need to collect several or hundreds of intensity images, which usually introduce motion artifacts when shooting fast-moving targets, leading to a decrease in image quality. Hence, a quantitative real-time phase microscopy (qRPM) for extended depth of field (DOF) imaging based on 3D single-shot differential phase contrast (ssDPC) imaging method is proposed in this research study. qRPM incorporates a microlens array (MLA) to simultaneously collect spatial information and angular information. In subsequent optical information processing, a deconvolution method is used to obtain intensity stacks under different illumination angles in a raw light field image. Importing the obtained intensity stack into the 3D DPC imaging model is able to finally obtain the 3D refractive index distribution. The captured four-dimensional light field information enables the reconstruction of 3D information in a single snapshot and extending the DOF of qRPM. The imaging capability of the proposed qRPM system is experimental verified on different samples, achieve single-exposure 3D label-free imaging with an extended DOF for 160 µm which is nearly 30 times higher than the traditional microscope system.

Fast Fourier ptychographic quantitative phase microscopy for in vitro label-free imaging.

Quantitative phase microscopy (QPM) is indispensable in biomedical research due to its advantages in unlabeled transparent sample thickness quantification and obtaining refractive index information. Fourier ptychographic microscopy (FPM) is among the most promising QPM methods, incorporating multi-angle illumination and iterative phase recovery for high-resolution quantitative phase imaging (QPI) of large cell populations over a wide field of-view (FOV) in a single pass. However, FPM is limited by data redundancy and sequential acquisition strategies, resulting in low imaging efficiency, which in turn limits its real-time application in in vitro label-free imaging. Here, we report a fast QPM based on Fourier ptychography (FQP-FPM), which uses an optimized annular downsampling and parallel acquisition strategy to minimize the amount of data required in the front end and reduce the iteration time of the back-end algorithm (3.3% and 4.4% of conventional FPM, respectively). Theoretical and data redundancy analyses show that FQP-FPM can realize high-throughput quantitative phase reconstruction at thrice the resolution of the coherent diffraction limit by acquiring only ten raw images, providing a precondition for in vitro label-free real-time imaging. The FQP-FPM application was validated for various in vitro label-free live-cell imaging. Cell morphology and subcellular phenomena in different periods were observed with a synthetic aperture of 0.75 NA at a 10× FOV, demonstrating its advantages and application potential for fast high-throughput QPI.

Fast digital refocusing Fourier ptychographic microscopy method based on convolutional neural network.

Fourier ptychographic microscopy (FPM) is used to achieve high resolution and a large field of view. However, traditional FPM image reconstruction methods often yield poor image quality when encountering out-of-focus issues during reconstruction. Therefore, this study proposes a defocus-distance regression network based on convolutional neural networks. In an experimental validation, the root-mean-square error calculated from 1000 sets of predicted and true values was approximately 6.2 µm. The experimental results suggest that the proposed method has good generalization, maintains high accuracy in predicting defocus distances even for different biological samples, and extends the imaging depth-of-field of the FPM system by a factor of more than 3.

Fourier ptychographic microscopy image stack reconstruction using implicit neural representations

Image stacks provide invaluable 3D information in various biological and pathological imaging applications. Fourier ptychographic microscopy (FPM) enables reconstructing high-resolution, wide field-of-view image stacks without z-stack scanning, thus significantly accelerating image acquisition. However, existing FPM methods take tens of minutes to reconstruct and gigabytes of memory to store a high-resolution volumetric scene, impeding fast gigapixel-scale remote digital pathology. While deep learning approaches have been explored to address this challenge, existing methods poorly generalize to novel datasets and can produce unreliable hallucinations. This work presents FPM-INR, a compact and efficient framework that integrates physics-based optical models with implicit neural representations (INRs) to represent and reconstruct FPM image stacks. FPM-INR is agnostic to system design or sample types and does not require external training data. In our experiments, FPM-INR substantially outperforms traditional FPM algorithms with up to a 25-fold increase in speed and an 80-fold reduction in memory usage for continuous image stack representations.